Multiple exposure methods and apparatus for electronic cameras

ABSTRACT

A method for capturing multiple sets of image data with an electronic camera having a shutter and an electronic shutter for selectively allowing light to reach an image sensor comprises opening the shutter and the electronic shutter, allowing light to reach the image sensor for a first exposure time, closing the electronic shutter, reading out pixel data captured during the first exposure time, allowing light to reach the image sensor for a second exposure time, and, reading out pixel data captured during the second exposure time. The method may be used to obtain multiple differently exposed images of a scene for combination into a high dynamic range image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/614,851 filed 9 Nov. 2009, which is a continuation of U.S. patentapplication Ser. No. 11/459,633 filed on 24 Jul. 2006 and issued as U.S.Pat. No. 7,616,256 on 10 Nov. 2009, which is a continuation-in-part ofU.S. patent application Ser. No. 11/236,155 filed on 26 Sep. 2005, whichclaims the benefit under 35 U.S.C. §119 of U.S. patent application No.60/663,245 filed on 21 Mar. 2005, all of which are entitled MULTIPLEEXPOSURE METHODS AND APPARATUS FOR ELECTRONIC CAMERAS and are herebyincorporated herein by reference.

TECHNICAL FIELD

The invention relates to electronic cameras, and particularly to methodsand electronic camera apparatus for capturing multiple exposures. Theinvention has application in capturing high dynamic range images.

BACKGROUND

Real world scenes can have contrast ratios of 50,000:1 between thebrightness of the brightest highlights and the darkest shadows. Manyconventional image formats and image rendering devices (such as digitalprojectors, computer monitors, and the like) are only capable ofreproducing contrast ratios of a few hundred to one. In such fileformats it is not uncommon for pixel brightness values to be specifiedusing one 8-bit number per colour.

High dynamic range (“HDR”) image formats permit recording contrastratios that are significantly greater than those of conventional 8-bitimage formats. For example, some HDR formats use 16 or 32 bits percolour to represent different levels of brightness.

One way to obtain image data for high dynamic range images is to acquiremultiple images with conventional imaging equipment at differentexposure levels. This technique is described, for example, in Debevec etal. Recovering High Dynamic Range Radiance Maps from Photographs,Proceedings of SIGGRAPH 97, Computer Graphics Proceedings, AnnualConference Series, pp. 369-378 (August 1997, Los Angeles, Calif.),Addison Wesley, Edited by Turner Whitted. ISBN 0-89791-896-7, which ishereby incorporated herein by reference.

A problem is that the multiple images need to be aligned with oneanother. This makes it necessary to use a tripod in most cases. Further,setting a camera to take multiple images appropriate for combinationinto an HDR image requires significant knowledge regarding theappropriate combination of exposures to use for each of the images.

HDR images are becoming mainstream. There is a need for methods andapparatus for easily acquiring HDR images.

SUMMARY OF THE INVENTION

This invention provides methods and apparatus for acquiring multipleexposures in electronic cameras. The methods and apparatus may beapplied to acquiring images that can be combined to yield an HDR image.Methods according to some embodiments of the invention can be practicedwith standard electronic camera hardware controlled by a controllerexecuting modified firmware.

One aspect of the invention provides a method of capturing images withan electronic camera having a shutter and an electronic shutter forselectively allowing light to reach an image sensor. The methodcomprises opening the shutter and the electronic shutter, allowing lightto reach the image sensor for a first exposure period, closing theelectronic shutter, reading out pixel data captured during the firstexposure period, allowing light to reach the image sensor for a secondexposure period, closing the shutter, and, reading out pixel datacaptured during the second exposure period either before or afterclosing the shutter. The pixel data is preferably read out after theshutter is closed. The pixel data for the first exposure period may beretained in a pixel store of the image sensor and the pixel data for thesecond exposure period may be retained in pixels of the sensor untilafter the shutter has closed. The method may be implemented by firmwarein a controller of an electronic camera.

Another aspect of the invention provides an electronic camera having ashutter, an electronic shutter, an array of light sensors and acontroller configured to cause the electronic camera to acquire multipleimages by: opening the shutter and the electronic shutter; allowinglight to reach the image sensor for a first exposure period; closing theelectronic shutter; reading out pixel data captured during the firstexposure period; allowing light to reach the image sensor for a secondexposure period; closing the shutter; and, reading out pixel datacaptured during the second exposure period either before or afterclosing the shutter. The shutter is held open for both the first andsecond exposure periods. The pixel data is preferably read out after theshutter is closed. The pixel data for the first exposure period may beretained in a pixel store of the image sensor and the pixel data for thesecond exposure period may be retained in pixels of the sensor untilafter the shutter has been closed by the controller.

Another aspect of the invention comprises an EPROM or othercomputer-readable medium carrying firmware instructions for execution bya controller of an electronic camera. The instructions, when executed bythe controller cause the controller to perform a method according to theinvention.

Another aspect of the invention provides a method of capturing an imagewith an electronic camera having a shutter for selectively allowinglight to reach an image sensor and means for selectively reading outpixel data from the image sensor. The method comprises opening theshutter, allowing light to reach the image sensor for a first exposureperiod, preserving a first portion of pixel data captured during thefirst exposure period, allowing light to reach the image sensor for asecond exposure period, and, reading out the first portion of pixel datacaptured during the first exposure period and pixel data captured duringthe second exposure period.

Other aspects of the invention and features of specific embodiments aredescribed below.

BRIEF DESCRIPTION OF DRAWINGS

In drawings which illustrate non-limiting embodiments of the invention:

FIG. 1 shows a sensor layout for an interline transfer CCD;

FIG. 2 is a block diagram of an electronic camera;

FIG. 3 is a flowchart illustrating the steps in a method according toone embodiment of the invention;

FIG. 4 is a flowchart illustrating the steps in a method according toanother embodiment of the invention;

FIGS. 5A and 5B show a high resolution CCD at two different times;

FIG. 6 is a flowchart illustrating the steps in a method according toanother embodiment of the invention; and,

FIG. 7 is a flowchart illustrating the steps in a method according toanother embodiment of the invention.

DESCRIPTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive, sense.

One aspect of the invention allows an electronic camera to be configuredto capture multiple exposures of a single image. The multiple exposuresmay be used to create a high dynamic range (HDR) image. Methods forcombining data from multiple exposures to yield a HDR image are known inthe art. For example, some such methods are described in Mann, S. et al.Being ‘undigital’ with digital cameras: Extending dynamic range bycombining differently exposed pictures, In Proc. IS&T 46th AnnualConference (May, 1995) pp. 422-428 which is hereby incorporated hereinby reference and in the Debevec et al. article referred to above.

Electronic cameras typically have a shutter which can be opened toselectively allow light to reach a light sensitive chip or closed toblock light from reaching the chip. When the shutter is open, a lensprojects an image onto the chip. The shutter may comprise a mechanicalshutter, for example.

The chip measures light intensity at a number of pixel locations. Valuesrepresenting the intensity at each of the pixels can be read out andstored in a memory. Light sensitive chips also typically include meansfor selecting the amount of time the light sensitive elements collectlight before the data stored therein is read out. Such means can act asan “electronic shutter”. For example, many CCD chips include a controlthat triggers reading out of pixel data from light-sensitive pixels intovertical data registers that are shielded from light. The inventor hasdetermined that during a single period while the mechanical shutter isheld open, the electronic shutter may be operated to achieve multipleexposures of a single image.

FIG. 1 shows a sensor layout for an interline transfer charge coupleddevice (“CCD”) 10 of the type used in many electronic cameras. CCD 10comprises a plurality of imaging regions 12 which comprise lightsensitive elements, and storage regions which comprise verticalregisters 14. In the illustrated embodiment, imaging regions 12alternate with vertical registers 14 in a horizontal direction. Imagingregions 12 each comprise a plurality of pixels 16. Each pixel 16comprises a sensor that stores charge. The amount of stored chargedepends upon the number of photons detected by the sensor. As indicatedby the arrows in FIG. 1, after an image is exposed, pixel datacharacterizing the charges stored in pixels 16 are shifted to verticalregisters 14.

Vertical registers 14 (which are sometimes referred to as “columnregisters”) may be covered with an opaque mask layer (e.g. with stripsof aluminum) to prevent incoming light from striking vertical registers14. Vertical registers 14 preserve the charges' characteristics as thepixel data is shifted down vertical registers 14 into a horizontalregister 18. (The terms “vertical”, “horizontal” and “down” used in thisdescription refer to the orientation of the elements shown in theFigures, rather than to any particular physical orientation.) The pixeldata is generally read out from horizontal register 18 to ananalog-to-digital converter through one or more amplifiers and/or othersignal conditioning circuits (not shown).

FIG. 2 shows an electronic camera 20 equipped with CCD 10 of FIG. 1.Camera 20 comprises a mechanical shutter 22 for selectively allowinglight from an image to reach CCD 10. When mechanical shutter 22 is open,the incoming light is focused by a lens (not shown) onto CCD 10. Acontroller 24 controls the opening and closing of shutter 22 by means ofa shutter control line 26. Controller 24 also controls the clocking ofvertical registers 14 and horizontal register 18 of CCD 10 by means of aCCD control line 28. Controller 24 may comprise, for example, amicroprocessor running software (e.g. firmware 25).

Controller 24 receives pixel data from CCD 10 by means of data line 30.Controller 24 may store the pixel data in a memory 32, display an imagebased on the pixel data on a view screen 34, or both. Camera 20 alsocomprises an interface 35 for allowing a user to interact withcontroller 24. Interface 35 includes a shutter release 36. Shutterrelease 36 may be triggered by a timer, an electronic signal, a shutterrelease button or the like.

FIG. 3 shows a method 100 for capturing multiple exposures of a singleimage using CCD 10 of FIG. 1. The mechanical shutter is open at thestart of method 100. In block 102, method 100 sets the camera's lens toan appropriate aperture. The aperture may be set by the camera'sexposure control circuit. A wide variety of suitable systems for settingthe shutter aperture in digital cameras are known in the art. Suchsystems may set the aperture to a user-determined value or may set theaperture according to an algorithm based upon detected light levels.

At block 104 any charge stored in pixels 16 is cleared, and a firstexposure begins. At block 106 the camera's exposure timer counts down apredetermined time period for the first exposure. After thepredetermined time period for the first exposure has elapsed, the chargestored in pixels 16 is shifted into vertical registers 14 at block 108.

Shifting of the charge from pixels 16 to vertical registers 14 at block108 simultaneously ends the first exposure and begins a second exposure.At block 110 the camera's exposure timer counts down a predeterminedtime period for the second exposure. After the predetermined time periodfor the second exposure has elapsed, the mechanical shutter is closed atblock 112, thereby ending the second exposure.

At block 114, pixel data from the first exposure (i.e. the charge frompixels 16 which was shifted into vertical registers 14 at block 108) isshifted down vertical registers 14 to horizontal register 18. At block116 the pixel data from the first exposure is read out from horizontalregister 18.

At block 118, the charge stored in pixels 16 from the second exposure isshifted into vertical registers 14. At block 120, pixel data from thesecond exposure is shifted down vertical registers 14 to horizontalregister 18. At block 122 the pixel data from the second exposure isread out from horizontal register 18. At block 124 method 100 ends. Inor after block 124, the mechanical shutter may be opened again to readythe camera for capturing the next image.

The exposures for the first and second exposures are different. This maybe achieved by making the time period for the first exposure differentfrom the time period for the second exposure. The predetermined timeperiod for the first exposure is preferably longer than thepredetermined time period for the second exposure. For example, thefirst exposure may last for ⅛ second and the second exposure may lastfor 1/125th second. The first and second exposure times may depend onminimum and maximum brightnesses of the image being captured and otherfactors such as the aperture setting. The aperture setting may bechanged between the first and second exposures.

Although two exposures are generally sufficient to produce a HDR image,method 100 described above may be modified to allow the camera tocapture more than two exposures. For example, FIG. 4 shows a method 200which may be used to allow an electronic camera to capture threeexposures of a single image during a period while a mechanical shutterremains open. The steps carried out at blocks 202 to 210 of method 200are the same as those carried out at blocks 102 to 110 of method 100described above.

During the predetermined time period for the second exposure (block210), pixel data from the first exposure is shifted down verticalregisters 14 to horizontal register 18 at block 212. At block 214 thepixel data from the first exposure is read out from horizontal register18. Thus, the steps of blocks 212 and 214 may be carried outsimultaneously with the step of block 210. At block 216 the chargestored in pixels 16 is shifted into vertical registers 14, therebyending the second exposure and starting a third exposure.

At block 218 the camera's exposure timer counts down a predeterminedtime period for the third exposure. After the predetermined time periodfor the third exposure has elapsed, the mechanical shutter is closed atblock 220, thereby ending the third exposure.

At block 222, pixel data from the second exposure is shifted downvertical registers 14 to horizontal register 18. At block 224 the pixeldata from the second exposure is read out from horizontal register 18.At block 226, the charge stored in pixels 16 from the third exposure isshifted into vertical registers 14. At block 228, pixel data from thethird exposure is shifted down vertical registers 14 to horizontalregister 18. At block 230 the pixel data from the third exposure is readout from horizontal register 18. At block 232 method 200 ends. Themechanical shutter may be opened again in or after block 232 to readythe camera for capturing the next image.

Method 200 has the feature that, when used in a camera that incorporatesa typical light sensor chip, it requires the readout of some data whenthe mechanical shutter is open. This can cause some “smearing” of thedata being read out, especially in parts of the sensor that are exposedto high levels of illumination. In some embodiments, such smearing iscompensated for by comparing image data for the first exposure, which isread out while the mechanical shutter remains open, to image data forthe second and/or third exposures, which can be read out after themechanical shutter has been closed. In other embodiments of theinvention, data corresponding to columns from the first exposure thatare in shadow, and therefore not too much affected by smearing, areselected and data from those columns is used to gain detail in shadowportions of an HDR image that is obtained by combining image data fromthe second and third exposures. In other embodiments the image data fromthe first exposure is used to contribute to the HDR image withoutcompensating for smearing. In some such embodiments, data from the firstexposure may be weighted differently in creating the HDR image than datafrom the second and/or third exposures.

A controller in a camera may be programmed to combine the two or moreexposures to yield HDR image data and to store the HDR image data in anysuitable HDR format. Where this is done, it can be appreciated that HDRimages can be obtained in a way that is essentially transparent to auser.

As will be appreciated by one skilled in the art, methods according toembodiments of the invention can be implemented in electronic cameras byproviding the cameras with modified firmware without the need for anynew hardware.

Some “high resolution” cameras having CCD chips employ sequentialreadout techniques when shifting pixel data to the vertical registers. Ahigh resolution camera may have, for example, a resolution of 3megapixels or more. Some sequential readout techniques involve shiftingdata from multiple pixels into a single charge storing region of avertical register over multiple clock cycles. The pixels may be dealtwith in portions, with each portion of the pixels comprising one pixelassociated with each of the charge storing regions of the verticalregisters. The number of portions of pixels is determined by the numberof pixels associated with each charge storing region.

FIGS. 5A and 5B show an example of a CCD 50 of a high resolution camerawhich employs a sequential readout technique in successive clock cycles.CCD 50 is similar to CCD 10 of FIG. 1, and comprises alternating imagingregions 52 and vertical registers 54. Each imaging region 52 comprises aplurality of pixels 56. Each pixel 56 comprises a sensor that stores anamount of charge determined by the number of photons detected by thesensor. Charge collected by pixels 56 may be shifted to verticalregisters 54, then down into a horizontal register 58 for read out.

Each vertical register 54 comprises a plurality of charge storingregions 60 capable of storing charge from pixels 56. In the illustratedexample, each charge storing region 60 has two pixels 56A and 56Bassociated therewith. Pixels 56A collectively comprise a first portion,and pixels 56B collectively comprise a second portion. Charge frompixels 56A of the first portion is simultaneously shifted to verticalregisters 54 at a first time, as shown in FIG. 5A. Charge from pixels56B of the second portion is simultaneously shifted to verticalregisters 54 at a second time, as shown in FIG. 5B.

When a camera having a CCD which employs sequential readout techniquesis operated to capture a single exposure, pixel data from a firstportion of pixels is shifted into the charge storing regions of thevertical registers after the exposure period ends. The pixel data isthen read out before pixel data from a next portion of the pixels isshifted into the charge storing regions, and the process repeats untilpixel data from all portions of the pixels has been read out. Theshutter is typically closed while the pixel data is being read out.

When a camera having a CCD which employs sequential readout techniquesis operated to capture multiple exposures according to embodiments ofthe invention, it may not be desirable to sequentially shift and readout pixel data from the first exposure period for all portions of thepixels. Waiting to read out all portions of the pixel data may delay thestart of the second exposure period.

In some embodiments of the invention this problem may be addressed byshifting and reading out pixel data from the first exposure period foronly the first portion of the pixels, and interpolating pixel data forthe remaining portions of the pixels. It may be desirable for the firstexposure period to be longer than the second exposure period in somesuch embodiments.

In other embodiments of the invention this problem may be addressed byshifting pixel data from all portions of the pixels into the chargestoring regions of the vertical registers and reading out combined pixeldata from the first exposure period.

In such embodiments, pixel data from each pixel associated with a chargestoring region may be approximated or calculated from the charge readout from that charge storing region. For example, if pixel data iscombined by summing pixel data from a plurality of pixels to one chargestoring region, pixel data for each of the pixels associated with thatcharge storing region may be determined by dividing the charge read outfrom each charge storing region by the number of portions of pixels. Ifpixel data is combined using a different technique, the pixel data maybe separated from the charge read out of the common charge storingregion using a corresponding technique.

Methods according to example embodiments of the invention are describedbelow with reference to FIGS. 6 and 7. The examples described in FIGS. 6and 7 contemplate a CCD wherein two pixels are associated with eachcharge storing region, such that the pixels are divided into twoportions, but it is to be understood that methods according to theinvention could be practice with any number of portions of pixels.

FIG. 6 shows a method 300 of capturing multiple exposures with a camerahaving a CCD which employs sequential readout techniques according toone embodiment of the invention. The steps carried out at blocks 302 to316 of method 300 are similar to those carried out at blocks 102 to 116of method 100 described above, except that at block 308 only the firstportion of pixels from the first exposure are shifted to the verticalregisters. At block 317, the second portion of pixels from the firstexposure are interpolated based on the readout of the first portion ofpixels at block 316. At block 318 the first portion of pixels from thesecond exposure are shifted to the vertical registers. The first portionof pixels from the second exposure are shifted to the horizontalregister at block 320 and read out from the horizontal register at block322. At block 324 the second portion of pixels from the second exposureare shifted to the vertical registers. The second portion of pixels fromthe second exposure are shifted to the horizontal register at block 326and read out from the horizontal register at block 328. At block 330method 300 ends. In or after block 330, the mechanical shutter may beopened again to ready the camera for capturing the next image.

FIG. 7 shows a method 400 of capturing multiple exposures with a camerahaving a CCD which employs sequential readout techniques according toanother embodiment of the invention. The steps carried out at blocks 402to 430 of method 400 are similar to those carried out at blocks 302 to330 of method 300 described above, except that at block 408 both thefirst and second portions of pixels from the first exposure are shiftedto the vertical registers and combined, and at block 417 the pixelswhich are read out from the horizontal register at block 416 are dividedby the number of portions of pixels (two in this example) to separatethe combined pixels.

Certain implementations of the invention comprise computer processorswhich execute software instructions which cause the processors toperform a method of the invention. For example, one or more processorsin a controller for an electronic camera may implement the methods ofFIG. 3, 4, 6 or 7 by executing software instructions in a program memoryaccessible to the processors. The invention may also be provided in theform of a program product. The program product may comprise any mediumwhich carries a set of computer-readable signals comprising instructionswhich, when executed by a data processor, cause the data processor toexecute a method of the invention. Program products according to theinvention may be in any of a wide variety of forms. The program productmay comprise, for example, physical media such as magnetic data storagemedia including floppy diskettes, hard disk drives, optical data storagemedia including CD ROMs, DVDs, electronic data storage media includingROMs, flash RAM, or the like. The computer-readable signals on theprogram product may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

As will be apparent to those skilled in the art in the light of theforegoing disclosure, many alterations and modifications are possible inthe practice of this invention without departing from the spirit orscope thereof. For example:

-   -   Any shutter device capable of selectively allowing light to        reach or blocking light from reaching a sensor array may be used        in place of mechanical shutter 22. The mechanical nature of        shutter 22 is optional.        Accordingly, the scope of the invention is to be construed in        accordance with the substance defined by the following claims.

What is claimed is:
 1. A method of capturing an image with an electroniccamera having a shutter for selectively allowing light to reach an imagesensor and readout circuitry for selectively reading out pixel data fromthe image sensor, the method comprising: capturing at least threesuccessive exposures of a single image with the image sensor during aperiod while the shutter remains open, wherein the at least threesuccessive exposures comprise a first exposure, a second exposurestarting once the first exposure ends, and a third exposure startingonce the second exposure ends; reading out data from the at least threesuccessive exposures; and combining the data from the at least threeexposures to yield high dynamic range image data wherein a setting ischanged between at least two of the at least three successive exposures.2. The method according to claim 1 comprising reading out data from thefirst exposure while capturing another exposure.
 3. The method accordingto claim 2 comprising compensating for smearing of data from the firstexposure by comparing pixel data from the first exposure tocorresponding pixel data for one or more subsequent exposures.
 4. Themethod according to claim 2 wherein combining the data comprisesweighting data from the first exposure differently than data from one ormore subsequent exposures.
 5. The method according to claim 1 whereineach of the at least three exposures comprise periods of differentlengths.
 6. The method according to claim 1 wherein the setting is anaperture setting.
 7. A program product comprising a non-transitorymedium carrying computer-readable instructions which, when executed by aprocessor in a controller for a camera, cause the controller to controlthe camera to perform a method according to claim
 1. 8. An electroniccamera comprising a controller having a processor and a memory storingcomputer-readable instructions which, when executed by the processor,cause the controller to control the camera to perform a method accordingto claim
 1. 9. An electronic camera comprising: an image sensor; ashutter for selectively allowing light to reach the image sensor;readout circuitry for selectively reading out pixel data from the imagesensor; and, a controller configured to control the shutter and thereadout circuitry, the controller comprising a processor and a memoryhaving computer-readable code embodied therein which, when executed bythe processor, causes the controller to: capture at least threesuccessive exposures of a single image with the image sensor during aperiod while the shutter remains open, wherein the at least threesuccessive exposures comprise a first exposure, a second exposurestarting once the first exposure ends, and a third exposure startingonce the second exposure ends; read out data from the at least threesuccessive exposures; and combine the data from the at least threeexposures to yield high dynamic range image data wherein the controlleris configured to change a setting between at least two of the at leastthree successive exposures.
 10. The camera according to claim 9 whereinthe controller is configured to read out data from the first exposurewhile capturing another exposure.
 11. The camera according to claim 10wherein the controller is configured to compensate for smearing of datafrom the first exposure by comparing pixel data from the first exposureto corresponding pixel data for one or more subsequent exposures. 12.The camera according to claim 10 wherein the controller is configured tocombine the data by weighting data from the first exposure differentlythan data from one or more subsequent exposures.
 13. The cameraaccording to claim 9 wherein each of the at least three exposurescomprise periods of different lengths.
 14. The camera according to claim9 wherein the setting is an aperture setting.